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Executive Summary 



Beginning in 2010, The Evaluation Center at the University of Colorado Denver was contracted by 
Denver Public Schools (DPS), Colorado Education Association (CEA), A+ Denver, and Denver Classroom 
Teachers Association (DCTA) to provide external evaluation services to study the Innovation schools in 
Denver. 

The report of the first year's study described the initial implementation of eight schools granted 
Innovation status in DPS between February 2008 and May 2010. This follow-up report examines how 
Innovation status has affected the school climate, workforce composition, and student outcomes at the 
initial eight schools and eleven additional schools granted Innovation status between May and August 
2011. The evaluation design was structured to examine these factors based on a review of school 
reform literature and a theory of change developed by the stakeholders. Results included these 
findings: 

School Climate and Culture 

Innovation school respondents scored higher on the Climate Survey than those in Comparison schools 
on all measures. The largest differences were on the scales related to Decision Making and Ownership, 
while the groups were most similar on the Collaborative Environment scale. 

Further analyses indicated those respondents in Cohort One (the first eight Innovation schools) scored 
higher on the Climate Survey than respondents from Cohort Two (the next group of eleven schools). 
Both groups from Innovation schools scored higher than those from Comparison schools; differences 
were statistically significant. These results suggest that it may take time for Innovation status to affect a 
school's climate and culture. However, it may also reflect fundamental differences among the schools. 

Workforce Profile 

Innovation schools as a group were found to have teachers with less experience than Comparison 
schools. However, schools having Innovation status for a longer period of time had higher average 
teaching experience than schools more recently attaining Innovation status. Trends showed experience 
levels declining for all groups between 2007-08 and 2010-11 and increasing in 2011-12. The exception is 
a decrease in average teaching experience for Cohort Two schools for 2011-12, the first year of 
Innovation status. A contributing factor to this decrease may be that six Cohort Two schools were new. 

In 2011-12, more than two-thirds (68%) of principals and assistant principals at Innovation schools were 
reported to be in their first or second year as school leaders in DPS. This pattern is evident to a slightly 
lesser extent in Comparison schools where 56% of principals were in their first or second year. Over 
time, Innovation and Comparison schools both show declining levels of principal experience. Overall in 
DPS, 37% of principals were new in 2011-12 and 34% in 2010-11, which may indicate other factors 
beyond Innovation status are impacting the high turnover rates for principals in this district. 
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Teachers in Innovation and Comparison schools were found to have similar education level profiles 
based on the highest degree earned. While Comparison schools had slightly more teachers with 
Masters degrees (51%) than Cohorts one and two of Innovation school groups (48% and 45%, 
respectively), these differences were not statistically significant. 

Teacher turnover was higher in Innovation schools in relation to Comparison schools and DPS as a 
whole, although trends over time appeared consistent across the groups. The higher rates of turnover 
may be contributing to the decreasing experience levels because newly hired teachers are more likely to 
be teachers new to the profession, especially since Innovation schools are not required to accept in- 
district transfers of more experienced teachers. Principal experience levels are also likely affected by 
the turnover rates among those serving in leadership roles. 

Student Achievement 

The 2012 Transitional Colorado Assessment Program (TCAP) results showed Innovation schools typically 
demonstrated higher growth than the state median but lower levels of proficiency than statewide 
averages in all subjects with a few exceptions. When TCAP results were compared to DPS as a whole, 
Cohort One Innovation schools were below the district's level of proficiency; however, their proficiency 
levels were higher than those in Comparison schools. 

Overtime, student proficiency levels have shown gradual improvement in Innovation schools. However, 
the clear trajectory of improvement in Innovation schools is mirrored in district-wide improvement 
trends and was evident in most schools before they attained Innovation status so, therefore, cannot be 
solely attributed to this factor. For Innovation high schools, ACT composite scores showed little 
variation over time while Comparison high schools showed a small increase in ACT scores. 

Conclusions and Questions for Further Exploration 

In this year's study, teachers at schools with Innovation status for a longer period of time reported their 
schools were characterized by the attributes of the "empowerment equation" identified by DPS 
stakeholders. While this finding is hopeful, it is unclear if these attributes were already present in the 
culture of the schools before seeking Innovation status or if Innovation status enhanced their work in 
any way. 

A complication to measuring the effectiveness of Innovation schools is the high turnover rates for both 
teachers and principals. Lack of continuity in both leadership and the workforce at Innovation schools 
may be interfering with progress that might otherwise be expected. What factors (including Innovation 
status) are influencing these high turnover rates? How can DPS support a more stable workforce in 
Innovation schools? 

Because rates of student achievement were improving in DPS overall during this time period, many 
reform strategies may be effectively improving student outcomes; Innovation status may be just one of 
many effective options. However, it may also be merely too soon to see differences between this 
strategy and competing initiatives for school reform. Examination of achievement trends over a longer 
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period of time is warranted for Innovation schools in comparison to those implementing other 
interventions intended to raise achievement levels. 

If the Innovation school theory of change is accurate, improved student outcomes should be evident in 
schools where autonomy in decision-making has been exercised for that purpose. This raises additional 
questions: Have Innovation schools actually implemented changes that would require Innovation status 
(as principals reported were planned in interviews in 2011)? Additionally, if changes have been made, 
how have they been directly related to improving professional practice (as opposed to more 
organizational re-structuring around budgets, schedules, and hiring practices)? These questions must 
be answered before an expectation of improved student outcomes can be examined in a meaningful 
way and distinguished from the expectation that student outcomes improve in all schools in DPS. 
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Introduction 



Beginning in 2010, The Evaluation Center in the School of Education and Human Development, at the 
University of Colorado Denver, was contracted through a collaborative effort by Denver Public Schools 
(DPS), Colorado Education Association (CEA), A+ Denver, and Denver Classroom Teachers Association 
(DCTA) to provide external evaluation services to study the Innovation schools in Denver. 

The first year of the Innovation schools study (2010-11) focused on answering the questions "What is 
happening in Innovation Schools? How is this different than what occurred prior to Innovation status?" 
These questions were examined for the cohort of the initial eight schools granted Innovation school 
status in DPS between February 2008 and May 2010. 1 (Please see a summary of the methods and 
results from the first year of this study in Appendix A.) 

The purpose of the second year of the study (2011-12) is to build on what was learned in year one, to 
better understand the changes that are occurring in Innovation schools, and to explore how these 
changes are related to workforce and student outcomes. The population of the schools has been 
expanded to include those schools granted Innovation status between May and August 2011. Schools 
included in this study are listed in Exhibit 1 below 2 . 

Exhibit 1: Innovation Schools in this Study 



School 


Date Innovation Status 


Bruce Randolph MS and HS 


February 2008 


Montclair School of Academics and Enrichment 


March 2009 


Manual High School 


March 2009 


Cole Arts and Sciences Academy 


August 2009 


Denver Green School 


April 2010 


Valdez Innovation School 


June 2010 


Whittier K-8 School 


September 2010 


Martin Luther King Jr. Early College MS and HS 


September 2010 


Noel Community Arts School 


May 2011 


Denver Center for International Studies at Ford and Montbello 


May 2011 


Collegiate Prep Academy 


June 2011 


High Tech Early College 


June 2011 


Denver Center for 21st Century Learning at Wyman MS and HS 


June 2011 


Godsman Elementary 


August 2011 


Green Valley Elementary 


August 2011 


McGlone Elementary 


August 2011 


Summit Academy 


August 2011 


Swigert-McAuliffe International School 


August 2011 


Vista Academy MS and HS 


August 2011 



1 Including Bruce Randolph School, granted Autonomous School status in February 2008 

2 Additional DPS schools have since been granted Innovation status but are not included in this study since it was 
considered too soon to expect changes to be evident; future studies may include additional Innovation schools. 
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Evaluation Framework 

The evaluation of the Innovation schools initiative is grounded in current school reform theory. The 
study is framed around factors which have been determined to be important in school reform by the 
Consortium on Chicago School Research (Bryk, Sebring, Allensworth, Luppescu, & Easton, 2010). The 
framework for the evaluation is summarized in Exhibit 2. 



Exhibit 2: Evaluation Framework 





Improve 
professional practice 






Y Improve ^1 

student 
L learning A 



Improve school 
culture 



A basic summary of what is included in each of these components is provided below. 

1. Improve school structures refers to changes in the way schools use time, engage in hiring, 
assess students, practice inclusive leadership, conduct personnel evaluations, and manage 
resources. 

2. Improve professional practice includes initiatives aimed at increasing teacher and principal 
effectiveness such as professional development, coaching, enhanced curricula, and 
qualifications for teachers and school leaders. 

3. Improve school culture refers to plans that enhance teacher, principal, and collective school 
efficacy beliefs; mutual trust among teachers, school administrators, parents and students; and 
the degree of academic emphasis within the school. 

4. Improve student learning is the central goal and stated purpose for Innovation schools and, 
therefore, is the anchor of the evaluation. 

This report includes analyses of teacher surveys designed to examine school climate and culture, teacher 
and principal data, and student achievement results, which relate to each of these components of 
school reform. 

Limitations 

Comparison schools were selected based on data for the initial cohort of eight Innovation schools and 
may not be the best matches for schools attaining Innovation status in 2011. Additional comparison 
schools were not selected because of budget constraints. The Colorado assessment tests were modified 
for 2012 to align with new state standards and, therefore, may not be comparable to results for 
previous years. Response rates to the survey were low and, therefore, results should be viewed 
cautiously since responding teachers may not represent the entire population. 
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Innovation Schools: Theory of Change 

At the start of 2012, stakeholders from DPS, A+ Denver, CEA, and DCTA engaged in discussions with the 
goal of understanding what each stakeholder group expected to happen in schools granted Innovation 
status. Through these discussions, the partners developed general ideas around changes expected to 
take place in an Innovation school (e.g., in use of people, time, and money), as well as how autonomy 
would be experienced and exercised, and how capacity would influence this process. Based on these 
discussions, a general theory of change model was developed by the evaluation team and shared with 
the stakeholders (see Exhibit 3). 

Exhibit 3: Theory of Change for Innovation Schools 



Theory of Change for innovation Schools 



Some stakeholders believe that a change in instruction is 
necessary in order for student outcomes to improve. 

Other stakeholders believe that instruction does not 
necessarily have to change - if a school already has 
strong instruction then changes to people, time, and 
money may be adequate to improve student outcomes. 



Changes that may occur 
Innovation schools 



Use of people 



use of money 



Exercise of autonomy; 
Decisions about what 
changes will positively 
impact students 



Instructional 
Program 



Moderates effects of 
these changes 



J 



Moderates effects of 
these changes 



Capacity 



Student 
outcomes 



Capacity is they key driver of whether changes 
made by the school will positively influence student 
outcomes. 

It is expected that in schools with low capacity, 
changes under innovation will not improve student 
outcomes. 



It is expected that in schools with high capacity, 
changes under Innovation will improve student 
outcomes. 



Underlying beliefs: 

innovation created a space for change, where schools have the opportunity to behave differently than they have previously with regards to 
lheir use nf people lime, and money 

Innovation schools are expected to make some sort of change(s) in the way that they utilize people, time and money. 
The success of Innovation schools will depend largely on their capacity to choose and implement meaningful changes. 
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Based on this theory of change, nine constructs were operationalized to represent the beliefs of DPS 
stakeholders with regard to what would happen when schools were given Innovation status: 

• Climate of innovation and professional learning - teachers are provided with opportunities to 
expand their own knowledge and development, and work to improve their teaching through 
innovative and novel solutions. 

• Collaborative environment - teachers collaborate around instruction and curriculum. They 
collaborate between and within grade levels, plan together, and have common beliefs about 
students' needs. 

• Decision making - the decisions made at the school involve appropriate stakeholders, are 
thoughtful and explicit, and are aligned with the schools' mission. Teachers have autonomy to 
make decisions that impact their students. 

• Development of capacity -there is strong capacity related to the workforce and instructional 
program, which includes coordination of instructional programs so that they are coherent, PD 
for teachers that is meaningful and differentiated, and learning opportunities that are well 
aligned with the school's plan. 

• Sense of empowerment - the extent to which teachers have control over or are able to influence 
decisions at the school including those related to instruction (e.g., textbooks, curriculum, 
pedagogy), workforce (e.g., their own teaching assignments, new hires), and their belief that all 
students are competent learners. 

• Sense of ownership - teachers feel ownership over what happens in their classroom, their 
school, and for all students. 

• Pride and fulfillment in work - teachers feel like their school is a place they would send their 
own child, which they would recommend to others, and that they feel proud to work there. 

• Self-accountability - teachers feel responsible for the students they teach, other students in the 
school, and feel a commitment to helping the whole school succeed. 

• Commitment to high quality outcomes - there are expectations for high performance from 
administration to teachers, teachers to teachers, and teachers to students. 

These nine constructs represent a combination of what DPS stakeholders have described as an 
"empowerment equation" of what happens to schools that receive Innovation status, as well as 
considerations raised by other stakeholders about contextual features, which may moderate the impact 
of Innovation on the school. The nine constructs are represented visually in Exhibit 4. 
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Exhibit 4: Outcomes of Innovation Status 



Sense of 
Empowerment 



Sense of 
Ownership 



Pride and 
fulfillment in 
work 



jlr jjr 

■V Decisions ■■r 
Commitment .. 



Commitment 
to high quality 
outcomes 



Decisions 
aligned in the 
interests of 
students* 



Improved 
student 
outcomes* 



Self- 
Accountability 



"These pieces of the theory of change 
were not directly assessed in the 
Climate Survey 



Contextual Factors 
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Climate Surveys 



Methods 

Instrument Development 

Based on what was learned from the evaluation study in 2010-11 regarding culture and climate in the 
Innovation Schools and taking into account the theory of change, A+, CEA, DCTA, and DPS stakeholders 
made the decision to explore the culture and climate of Innovation schools using a self-developed 
instrument (as opposed to a standardized instrument) in order to examine the most relevant elements. 

Survey items were constructed around the nine constructs in the theory of change developed by 
stakeholders. The items were designed to explore the key constructs, which the groups had determined 
were related to changes theorized to occur and ultimately related to student outcomes. The total 
survey contained 68 questions divided into nine major content areas, as shown in Exhibit 5Exhibit . 

Exhibit 5: Survey Constructs and Items 



Scale Title 


Number of items 


Climate of innovation and professional learning 


8 


Collaborative environment 


6 


Decision making 


9 


Development of capacity 


10 


Sense of empowerment* 


16 


Sense of ownership* 


3 


Pride and fulfillment in work* 


6 


Self-accountability* 


7 


Commitment to high quality outcomes* 


3 



Five of the scales were directly reflective of the DPS theory of change (as designated by a * in the table 
above). The other four scales were developed based on stakeholder discussions regarding contextual 
factors that would moderate the likelihood of the other elements playing out in a successful way. 

Respondents 

DPS provided an email list of all teachers in Innovation schools, as well as the five matched Comparison 
schools, as of May 2012. (See Appendix B for a description of the process used to identify Comparison 
schools.) All schools granted Innovation status from February 2008 to August 2011 were included in the 
sample (a total of 19 schools). For additional analysis, the Innovation schools were divided into two 
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cohorts 3 to see if differences were evident for those who had been Innovation schools for a longer time. 
Teachers were identified by their job title, so the final list of teachers included 805 individuals. There 
were 229 respondents who fully completed the survey and an additional 31 who partially completed the 
survey, for a response rate of 32% (260 out of 805). There were 84 respondents from Comparison 
schools, and 176 respondents from Innovation schools. Respondents represented all five Comparison 
schools and 18 of the 19 Innovation schools. 

Analysis 

Scale validation 

Items were grouped based on the category assigned during item development (i.e., face validity). Each 
of the nine scales had between 3 and 16 items initially. An alpha coefficient was calculated separately 
for each scale to determine how well the items fit together in terms of measuring a similar construct. 4 
The alpha scores for the nine scales are shown below: 

Exhibit 6: Alpha values for each subscale 



Scale Title 


Alpha coefficient 


# of items 


Climate of innovation and professional learning 


0.93 


8 


Collaborative environment 


0.90 


6 


Decision making 


0.93 


9 


Development of capacity 


0.90 


10 


Sense of empowerment 


0.90 


15 


Sense of ownership 


0.78 


3 


Pride and fulfillment in work 


0.92 


6 


Self-accountability 


0.92 


5 


Commitment to high quality outcomes 


0.81 


3 



The statistics for each item were examined to determine whether the item had an acceptable 
correlation with the rest of the scale and whether the alpha coefficient would improve substantially if 
the item were deleted. Based on these analyses, three items were removed for the purposes of 
analysis. (See Appendix C for items that were removed.) 



3 Cohort One = 8 schools granted Innovation status from February 2008 to September 2010; Cohort Two = 11 
schools granted Innovation status from May to August 2011. 

4 Acceptable internal consistency reliability is 0.70. 
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Scale scores 

For each of the respondents, a total score was calculated by summing the items for each of the nine 
scales. Respondents were excluded for analyses of a particular scale if they had not answered all the 
items, reducing the sample by about 24 respondents, depending on the scale. 

Correlations between subscales 

The theory of change suggests there are a number of different domains pertinent to improved student 
outcomes (e.g., sense of empowerment, pride and fulfillment). In order to ensure that the different 
scales were in fact measuring different concepts, scores on each subscale were correlated with one 
another. Correlations ranged from 0.25 to 0.76, indicating the subscales were related but not identical 
constructs. The correlation matrix for the nine scales is shown in Exhibit 7. 5 

Exhibit 7: Pierson Correlation Coefficients, N = 181 





Innovation/Professional 
Learning 


Collaborative Environ 


Commitment to 
outcome 


Decision Making 


Capacity 


Pride and Fulfillment 


Self-Accountability 


Empowerment 


Ownership 


Innovation/Pro Learning 


1.00 


















Collaborative Environment 


0.64 


1.00 
















Commitment to Outcomes 


0.66 


0.47 


1.00 














Decision Making 


0.54 


0.33 


0.66 


1.00 












Capacity 


0.59 


0.50 


0.68 


0.52 


1.00 










Pride and Fulfillment 


0.44 


0.25 


0.58 


0.70 


0.50 


1.00 








Self-Accountability 


0.74 


0.59 


0.71 


0.42 


0.47 


0.38 


1.00 






Empowerment 


0.51 


0.26 


0.55 


0.76 


0.42 


0.55 


0.39 


1.00 




Ownership 


0.51 


0.33 


0.69 


0.76 


0.47 


0.65 


0.48 


0.70 


1.00 



Statistical comparisons 

Differences between respondents from Innovation schools and Comparison schools were examined 
using t-tests; differences between Cohort One, Cohort Two, and Comparisons school respondents were 
examined using analysis of variance (ANOVA) with Tukey post-hoc analysis. 



5 This correlation matrix relies on data for those persons who had complete sets of data on all nine scales for the 
analysis. 
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Results 

Innovation school respondents scored higher than those in Comparison schools on all measures on the 
Climate Survey; results were statistically significant. 6 The largest differences were on the Decision 
Making and Ownership scales, while the groups were most similar on the Collaborative Environment 
scale. Although the differences cannot be attributed to Innovation status, the schools scored higher 
than would be expected merely by chance. (See Appendix C for results by survey item.) 

Exhibit 8: Average Climate Survey Scores, by Scale 



Scale 


Comparison 


Innovation 


Difference 
(sorted high to low) 


Decision Making 


2.24 


2.88 


0.64** 


Ownership 


2.48 


3.07 


0.59** 


Commitment to Outcomes 


2.91 


3.4 


0.49** 


Empowerment 


2.35 


2.79 


0.44** 


Pride and Fulfillment 


2.47 


2.89 


0.42** 


Innovation and Professional Learning 


2.77 


3.18 


0.41** 


Capacity 


2.28 


2.65 


0.37* 


Self-Accountability 


3.02 


3.32 


0.30* 


Collaborative Environment 


2.74 


2.89 


0.15 



*p<-001, "p<.05 



Further analyses indicated the differences between Cohort One, Cohort Two, and Comparison schools 
were statistically significant. 7 Differences were greatest between Cohort One and Comparison schools, 
as shown in Exhibit 9; Cohort One teachers also had higher scores than those in Cohort Two. 



Exhibit 9: Average Climate Survey Scores by Scale and Cohort 



Scale 


Comparison 


Innovation 
Cohort One 


Innovation 
Cohort Two 


Diff 
Comp & 
Cohort One 


Diff 
Comp & 
Cohort Two 


Diff 
Cohort One & 
Cohort Two 


Innovation/Professional Learning 


2.77 


3.25 


3.05 


0.48** 


0.28* 


0.19 


Collaborative Environment 


2.74 


2.92 


2.85 


0.18 


0.11 


0.07 


Commitment to Outcomes 


2.91 


3.51 


3.24 


0.59** 


0.33* 


0.27* 


Decision Making 


2.24 


3.00 


2.70 


0.76** 


0.46** 


0.30* 


Capacity 


2.28 


2.75 


2.49 


0.46** 


0.20 


0.26* 


Pride and Fulfillment 


2.47 


3.02 


2.69 


0.56** 


0.22 


0.33* 


Self-Accountability 


3.02 


3.40 


3.18 


0.39** 


0.16 


0.22 


Empowerment 


2.35 


2.88 


2.63 


0.53** 


0.29* 


0.25* 


Ownership 


2.48 


3.20 


2.89 


0.73** 


0.41* 


0.32* 



*p < .001, **p <.05 



6 1 (179) = 6.71, p < .001, Cohen's d = 1.07, a large effect 

7 F (2) = 24.90, p < .001, partial eta squared = .219, a large effect 
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As shown in the summary in Exhibit 10 below, Innovation schools in both cohorts had higher scores than 
Comparison schools. Differences were greatest for those who have been Innovation schools longer. 
These results suggest it may take time for Innovation status to affect a school's climate and culture. 
However, it may also reflect fundamental differences between the cohorts of schools. 

Exhibit 10: Average Climate Survey Score by Scale and Cohort 



Innovation and 
Professional 
Learning 

4^ 



Ownership 



Empowerment 



Collaborative 
Environment 



Self- 
Accountability 




•Comparison 
•Innovation Cohort 1 
Innovation Cohort 2 



Commitment to 
Outcomes 



Decision Making 



Pride an 
Fulfillment 



Capacity 



Summary - Climate Survey Results 

The goal of this survey was to explore the constructs outlined in the theory of change and to provide 
preliminary evidence of differences in Innovation and Comparison schools in these areas. The theory of 
change suggests that greater empowerment of schools (through receiving Innovation status) will lead to 
other positive changes in the school (including climate and behavior), which should result in improved 
outcomes for students. 



Beyond the general expectation that Innovation schools would score higher than Comparison schools in 
these areas, there were no specific expectations about how large the differences would be or what 
absolute level should be considered 'good enough' on any of the scales. What may be most valuable 
will be to have ongoing conversations about the results and implications for Innovation schools. (See 
Appendix D for suggested questions for discussion concerning Climate Survey results.) 
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Workforce Profile 



Because Innovation status allows schools greater autonomy and flexibility in operational decision- 
making such as hiring practices, changes in the workforce at these schools were examined. 

Methods 

The DPS Human Resources Department provided workforce data for teachers (n = 1,404) and principals 1 
(n = 95) at Innovation and Comparison schools from 2006-07 to 2011-12. Innovation schools were 
divided into the same two cohorts as used for the Climate Survey analyses; the same five Comparison 
schools were also used. Workforce analyses were designed to address three questions: 

Ql: How does the level of teacher and principal experience compare in Innovation and Comparison 
schools? 

Average total years of teaching experience were calculated for all teachers 9 in Innovation schools by 
cohorts and for Comparison schools. This included their experience both in DPS and in other districts. 
Differences between groups were compared for 2011-12 data using Kruskall-Wallis analysis of 
variance. 10 Trends over time were graphed. Average principal experience for those in leadership roles 
in 2011-12 also was examined using data for only DPS because experience outside of DPS was not 
reported for all individuals. 

Q2: Do Innovation schools have teachers with more education than Comparison schools? 

Percentages of teachers earning Bachelors, Masters, and Doctorate degrees as their highest level of 
education were calculated for Innovation schools by cohorts and for Comparison schools. For 2011-12 
data, Chi-square analysis was conducted to examine if observed rates of educational attainment were 
significantly different than expected outcomes for each group. 

Q2: How does being an Innovation school affect teacher turnover? 

Turnover rates for Innovation schools by cohorts and for Comparison schools were calculated for the 
period of 2007-08 to 2011-12 using the formula used by the Colorado Department of Education (CDE). 11 
Results for 2011-12 were compared using Chi-square analysis to examine if observed turnover rates 
were significantly different than expected rates for each group. Turnover percentages for DPS as a 
district were downloaded from CDE for further comparison 12 . 



s Includes principals and assistant principals 

9 Teachers with less than .5 assignments were deleted to avoid duplication. 

10 Non-parametric statistical analyses were used for all workforce data analyses because samples were not 
normally distributed. 

11 The number of teachers who leave a school for any reason is divided by the number of teachers employed the 
prior year to calculate the turnover percentage. 

12 CDE website, http://www.cde.state.co.us/index_stats.htm 
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Results 

Teacher Experience 

In 2011-12, teachers in Innovation schools as a group were less experienced than their counterparts in 
Comparison schools. When differences were examined by cohorts, schools having Innovation status for 
a longer period of time had higher average teaching experience than schools more recently attaining 
Innovation status, as shown in Exhibit 11. The differences were statistically significant. 13 

Exhibit 11: Average Teaching Experience in 2011-12 





n 


Mean 


Std. Dev 


Innovation Cohort One 


288 


5.22 


6.72 


Innovation Cohort Two 


232 


3.77 


5.61 


Comparison 


273 


9.50 


9.30 



Over time, Innovation schools consistently had teachers with less teaching experience on average than 
Comparison schools, as shown in Exhibit 12. Trends show similar patterns of change in the groups with 
average teaching experience declining between 2007-08 and 2010-11 and increasing in the last year. 
The exception is a decrease in average teaching experience for Cohort Two schools for 2011-12, the first 
year of Innovation status. A contributing factor to this decrease may be that six Cohort Two schools 
were new and added to these data only in 2011-12; new schools may be more likely to hire newer 
teachers with less teaching experience. 

Exhibit 12: Average Years of Teaching Experience over Time 



12 



10 




•Innovation Cohort One 
•Innovation Cohort Two 
Comparison 



1 1 1 1 1 

2006-07 2007-08 2008-09 2009-10 2010-11 2011-12 



Kruskal-Wallis analysis of variance results: \ (2, N = 793) = 90.59, p < .001 
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Principal Experience 

In 2011-12, more than two-thirds (68%) of principals and assistant principals at Innovation schools were 
reported to be in their first or second year as school leaders in DPS, as shown in Exhibit 13. This pattern 
is evident to a slightly lesser extent in Comparison schools where 56% of principals were in their first or 
second year. However, overall in DPS, 37% of principals were new in 2011-12 and 34% in 2010-11, 
which may indicate other factors beyond Innovation status are impacting the high turnover rates for 
principals (e.g., transfers within the district, promotion to other administrative positions). 



Exhibit 13: Principal Experience 



Years DPS 


Innovation Cohort 


Innovation Cohort 


Comparison 


Total 


Principal Experience 


One 


Two 


1 


3 


11 


3 


17 


2 


7 


5 


6 


18 


3 


2 





3 


5 


4 


1 


1 


1 


3 


5 








1 


1 


6 





1 





1 


7 


1 


1 


2 


4 


8 


1 


1 





2 


9 


1 








1 


10 


1 








1 


12 





1 





1 


Total 


17 


21 


16 


54 



Overtime, Innovation and Comparison schools both show declining levels of principal experience, as 
shown in Exhibit 14. 

Exhibit 14: Average Years of Administrator Experience over Time 
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Teacher Education Levels 

Teachers in Innovation and Comparison schools were found to have similar education level profiles, as 
shown in Exhibit 15, which shows the highest degree earned for those teachers in these schools in 2011- 
12. While Comparison schools had slightly more teachers with Masters degrees (51%) than either 
Innovation school cohort groups (48% and 45%, respectively), these differences were not statistically 
significant (may be just chance). 14 These data were also examined for the five previous school years 
with similar results. 



Exhibit 15: Percentage of Teachers by Highest Degree Earned 
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14 Chi-square analysis results: \ (4, N = 791) = 1.52, p = .823 
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Teacher Turnover 

Teacher turnover was higher in Innovation schools in relation to Comparison schools and DPS as a 
district, as shown in Exhibit 16, although trends over time appeared consistent across the groups. 
Turnover rates for Cohort One Innovation schools ranged from 23% to 37%; Cohort Two schools 15 
ranged from a low of 20% in 2009-10 to a high of 46% turnover in 2011-12. Comparison schools ranged 
from 10% to 34% while turnover rates in DPS as a whole were more stable (range from 14% to 21%). 
(See results by year for each group in Appendix E.) 

When 2011-12 turnover rates were compared, differences between the Innovation schools by cohort 
and Comparison schools were found to be statistically significant (not just chance). 16 

Exhibit 16: Percent Teacher Turnover 
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Summary - Workforce Profile 

Innovation schools as a group were found to have teachers with less experience than Comparison 
schools. The higher rates of turnover may be contributing to the decreasing experience because newly 
hired teachers are more likely to be teachers new to the profession, especially since Innovation schools 
are not required to accept in-district transfers of more experienced teachers. Principal experience 
levels are also likely affected by the turnover rates among those serving in leadership roles. 



Results for Cohort Two do not include those six schools that were new in 2011-12. 
16 Chi-square analysis results: \ (2, N = 1,456) = 62.67, p < .001 
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School Achievement 



Methods 

Achievement results were examined using school-level data from the Transitional Colorado Assessment 
Program (TCAP) for 2012 and the Colorado Student Assessment Program (CSAP) for 2004-2011, as data 
were available for the schools. Results for Colorado administration of the ACT were also examined. For 
the analyses, Innovation schools were divided into four cohorts according to the year in which they were 
granted Innovation status, as shown in Exhibit 17. The same five Comparison schools used in the 
Climate Survey and Workforce analyses were included. 



Exhibit 17: Cohorts for Achievement Data Analyses 



Cohort 


School 


Academic Year Innovation 
Status Initiated 


Cohort One 


Bruce Randolph MS and HS 


2008-09 


Cohort Two 


Montclair, Manual HS, Cole 


2009-10 


] Cohort Three 


Denver Green, Valdez, Whittier, Martin Luther King Jr. Early College 


2010-11 


Cohort Four 


Noel, Denver Center for International Studies at Ford and Montbello, 
Collegiate Prep Academy, High Tech Early, Godsman Elementary, Green Valley 
Elementary, McGlone Elementary, Summit Academy, Swigert-McAuliffe 
International School, Vista Academy 


2011-12 



Student achievement analyses were designed to address three questions: 

Ql: How does performance in DPS Innovations schools compare to state averages/medians? 

The percentages of students scoring at proficient/advanced and the median growth percentiles for each 
TCAP/CSAP subject in 2012 were graphed in relation to the statewide average percentages and the state 
median. 

Q2: How does performance in Innovation schools compare to other schools? 

Achievement results for 2012 were summarized by TCAP/CSAP subjects by calculating an unweighted 
average 17 of school level proficient/advanced rates for Innovation and Comparison schools. Results 
were compared to the overall rate of proficiency in DPS. ACT composite scores were also compared for 
high schools by cohort. 

Q3: What is the trajectory of achievement scores in the Innovation schools? 

Data for proficiency and growth levels at Innovation schools by cohorts, for Comparison schools, and for 
DPS overall were graphed to show changes over time. These data were also graphed showing 
achievement trends before and after attaining Innovation status. Similar graphs were created for each 
high school's composite ACT scores over time. Trends were examined to determine whether changes 
were evident in outcomes following the attainment of Innovation status. 

17 This method was selected to provide a representation of school level performance (rather than using the 
percentage of individual students, which could over represent schools with large populations). 
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Results 

TCAP Comparison to State Results for Proficiency and Growth 

On state assessments in 2012, Innovation schools typically demonstrated higher growth than the state 
median but lower levels of proficiency in all subjects. Results are presented by TCAP subject areas as 
shown in Exhibits 18 - 20 below. 

Moth 

Two schools (Green Valley Elementary and Montclair) scored above the 2012 state average of 
proficient/advanced on math assessments (56%) and also showed higher growth than the state median. 
Three schools were below both the state average for proficiency and the state median for growth 
(Denver Center for 21rst Learning at Wyman, Vista, and Bruce Randolph). The remaining 13 schools had 
average scores above the state median for growth but below the state average for proficiency. 18 

Exhibit 18: 2012 Math - proficiency and growth 




School • 



Median growth percentile 

Bruce Randolph School 
Cole Arts And Science Academy 
Collegiate Preparatory Academy 
Deis At Montbello 

Denver Center For 21st Learning At Wyman 

Denver Green School 

Gods man Elementary School 

Green Valley Elementary School 

High Tech Early College 

Manual High School 

Martin Luther King Jr. Early College 

Mcglone Elementary School 

Montclair Elementary School 

Noel Community Arts School 

Summit Academy 

Valdez Elementary School 

Vista Academy 

Whittier K-8 School 



Complete math data were not available for Swigert-McAuliffe and DCIS at Ford. 
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Reading 

All Innovation schools were below the 2012 state average of proficient/advanced on reading 
assessments (69%). Six schools (Denver Center for 21st Century Learning at Wyman, Manual, Vista, DCIS 
at Montbello, Noel, and Martin Luther King) demonstrated lower growth than the state median while 11 
schools were at or above the state median growth. 19 

Exhibit 19: 2012 Reading - proficiency and growth 



100- 



D 

8 
1 
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School 



Median growth percentile 

Bruce Randolph School 
Cole Arts And Science Academy 
Collegiate Preparatory Academy 
Deis At Montbello 

Denver Center For 21st Learning At Wyman 

Denver G*een School 

Godsman Elementary School 

Gteen Valley Elementary School 

High Tech Early College 

Manual High School 

Martin Luther King Jr. Early College 

Mcglone Elementary School 

Montclair Elementary School 

Noel Community Arts School 

Summit Academy 

Valdez Elementary School 

Vista Academy 

WhittierK-8 School 



Complete reading data were not available for Swigert-McAuliffe, DCIS at Ford, and Summit. 
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Writing 

All Innovation schools were at or below the 2012 state average proficient/advanced on writing 
assessments (54%). Three were below the state median for growth (Manual, Denver Center for 21 ! 
Century Learning at Wyman, and Vista). The other 14 schools were at or above the state growth 
median. 20 

Exhibit 20: 2012 Writing - proficiency and growth 
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.State Average 
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Median growth percentile 



Bruce Randolph School 
Cole Arts And Science Academy 
Collegiate Preparatory Academy 
Deis At Montbello 

Denver Center For 21st Learning At Wyman 

Denver Green School 

Gods man Elementary School 

Green Valley Elementary School 

High Tech Early College 

Manual High School 

Martin Luther King Jr. Early College 

Mcglone Elementary School 

Montclair Elementary School 

Noel Community Arts School 

Summit Academy 

Valdez Elementary School 

Vista Academy 

WhittierK-8 School 



Complete reading data were not available for Swigert-McAuliffe, DCIS at Ford, and Summit. 
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TCAP Comparison to District Results and Selected Schools 

Schools that attained Innovation status from 2008 - 20 10 21 demonstrated lower levels of proficiency on 
2012 assessments than DPS as a whole. However, Innovation schools as a group had a higher rate of 
proficiency than the selected Comparison schools, as shown in Exhibit 21. 

Exhibit 21: 2012 School-Level Proficiency by Subject Area 

100 




I Comparison 
I Innovation 
DPS 



Math 



Reading 



Writing 



TCAP/CSAP Achievement over Time 

Percentage Proficient/Advanced 

Student achievement as indicated by the percentage of students scoring proficient/advanced on state 
assessments has shown gradual improvement over time in Innovation schools as cohorts. 22 This trend is 
consistent with the improvement in all DPS schools. However, the pattern in the Comparison schools is 
less consistent with the district's overall trend of improvement, showing nearly flat rates of 
performance. 

Exhibits 22 - 24 show achievement trends over time by subject for Innovation schools by cohort, for 
Comparison schools, and for DPS. (Achievement trends for individual schools are presented in Appendix 
F.) 



This excludes the cohort of schools attaining Innovation status during 2011-12 because they had less than a year 
of implementation at the time of the state assessments. 

22 While proficiency results are graphed for the cohort of schools attaining Innovation status at the beginning of 
2011-12, this information is presented only as baseline information. 
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Exhibit 23: Reading TCAP/CSAP results - percent of students proficient/advanced over time 



* * * Comparison * •'- -'- DPS District Average * * * Innovation 08-09 

■ Innovation 09-10 • • ♦ Innovation 10-11 • • ♦ Innovation 11-12 




03-04 04-05 05-06 06-07 07-08 08-09 09-10 10-11 11-12 

Academic Year 
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Exhibit 24: Writing TCAP/CSAP results - percent of students proficient/advanced over time 




03-04 04-05 05-06 06-07 07-08 08-09 09-10 10-11 11-12 

Academic Year 



Median Growth Percentiles 

As was seen in 2012 results, Innovation schools tend to show average academic growth above the state 
median (50 th percentile). While the growth percentile for all DPS schools has been nearly flat and 
consistent with the state median, growth in the Innovation school cohorts 23 has been more erratic, as 
shown in Exhibits 25 - 27. This may be due to differences among cohorts of students over the years, 
especially in schools with small populations where median scores are less reliable. 



Again growth results for the cohort of schools attaining Innovation status at the beginning of 2011-12 is 
presented only as baseline information. 
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Exhibit 25: Math TCAP/CSAP results - growth percentile over time 



Comparison 
Innovation 09-10 



DPS District Average 
Innovation 10-11 



Innovation 08-09 
Innovation 11-12 
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Exhibit 26: Reading TCAP/CSAP results - growth percentile over time 
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Innovation 09-10 



DPS District Average 
♦ Innovation 10-11 
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Exhibit 27: Writing TCAP/CSAP results - growth percentile over time 



Comparison 
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DPS District Average 
♦ Innovation 10-11 
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TCAP/CSAP Achievement Relative to Innovation Status 

For those seven Innovation schools having achievement results prior to Innovation status and at least 
one year after the implementation year, results indicated schools were showing improvements in 
student performance (cells shaded in green in Exhibit 28) both before and after Innovation status. 
However, it is too soon to draw conclusions based on these results, which include only a small number 
of schools for a limited time period. (For proficiency and growth results for individual schools by 
TCAP/CSAP subject area relative to their attainment of Innovation status, see Appendix G.) 

Exhibit 28: Proficiency Rates Relative to Innovation Status 



School 
(Years) 


TCAP/CSAP 
Subject 


Percent Proficient/Advanced (Change from Previous Year) 


2 years prior 
Innovation 


1 year prior 
Innovation 


Innovation 
year 


1 year after 
Innovation 


2 years after 
Innovation 


Bruce 
Randolph 

(07- 11) 


Math 


14 


16 (+2) 


13 (-3) 


17 (+4) 


18 (+1) 


Reading 


17 


23 (+6) 


26 (+3) 


33 (+7) 


31 (-2) 


Writing 


11 


12 (+1) 


16 (+4) 


14 (-2) 


16 (+2) 


Cole 25 
(09-12) 


Math 




21 


25 (+4) 


32 (+7) 


39 (+7) 


Reading 




22 


28 (+6) 


30 (+2) 


36 (+6) 


Writing 




13 


19 (+6) 


23 (+4) 


28 (+5) 


Manual 
(08-12) 


Math 


6 


6(0) 


12 (+6) 


14 (+2) 


5 (-9) 


Reading 


30 


38 (+8) 


39 (+1) 


33 (-6) 


31 (-2) 


Writing 


17 


19 (+2) 


17 (-2) 


21 (+4) 


13 (-8) 


Montclair 
(08-12) 


Math 


36 


53 (+19) 


56 (+3) 


59 (+3) 


58 (-1) 


Reading 


41 


48 (+7) 


54 (+6) 


64 (+10) 


64 (0) 


Writing 


22 


35 (+13) 


43 (+8) 


49 (+6) 


54 (+5) 


Martin 
Luther King 

(09-12) 


Math 


26 


21 (-5) 


21(0) 


21(0) 




Reading 


36 


44 (+8) 


41 (-3) 


39 (-2) 




Writing 


27 


24 (-3) 


24 (0) 


29 (+5) 




Valdez 
(09-12) 


Math 


23 


28 (+5) 


40 (+13) 


37 (-3) 




Reading 


26 


33 (+7) 


36 (+3) 


33 (-3) 




Writing 


22 


23 (+1) 


23 (0) 


17 (-6) 




Whittier 
(09-12) 


Math 


40 


33 (-7) 


41 (+8) 


42 (+1) 




Reading 


36 


41 (+5) 


42 (+1) 


54 (+8) 




Writing 


31 


25 (-6) 


38 (+13) 


42 (+5) 





Bruce Randolph is the only school with three years of achievement results; 2012 results were not used to allow 
for comparisons with other schools at similar points in time. 
25 Cole was closed during 2007-2008. 
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ACT Results 

For Innovation high schools (Bruce Randolph, Manual, Martin Luther King), ACT composite scores 
showed little variation over time, as shown in Exhibit 29. The average change between 2011 and 2012 
was - .24. Comparison schools showed a small increase over time (change between 2011 and 2012 was 
+.56). (See Appendix H for ACT results over time in relation to attainment of Innovation status.) 

Exhibit 29: Average Composite ACT Scores over time 
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Summary - School Achievement 

Student achievement data indicated Innovation schools are showing improvement in performance; 
however, there is insufficient data to connect these results to their status as Innovation schools. While 
it is hopeful that Innovation schools are demonstrating high rates of growth compared to the state 
median, similar growth rates were seen in Comparison schools. Although Innovation schools showed 
slightly higher rates of proficiency than Comparison schools, the small sample and limited time period 
make it too soon to be conclusive. The clear trajectory of improvement in Innovation schools is 
mirrored in district-wide improvement trends and was evident in most schools before they attained 
Innovation status so, therefore, cannot be solely attributed to this factor. It is likely that many other 
factors are ultimately influencing achievement levels. As Innovation school achievement is monitored 
over time, it may be useful to view Innovation status as one possible intervention and compare results 
to schools implementing other types of school reform to examine whether Innovation status is equally 
effective as other options. 
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Conclusions and Questions for Further Exploration 



In this year's study, teachers at schools with Innovation status for a longer period of time reported their 
schools were characterized by the attributes of the "empowerment equation" identified by DPS 
stakeholders. While this finding is hopeful, it is unclear if these attributes were already present in the 
culture of the schools before seeking Innovation status. In other words, were the teachers and leaders 
(especially those in the first group of Innovation schools) already empowered and engaged in strategies 
to improve student outcomes before they pursued Innovation status? If they were already exhibiting 
these constructs, did Innovation status facilitate or enhance their work in any way? 

A complication to measuring the effectiveness of Innovation schools is the high turnover rates for both 
teachers and principals. While this trend is also evident in DPS as a whole, lack of continuity in both 
leadership and the workforce at Innovation schools may be interfering with progress that might 
otherwise be expected. If Innovation schools are exercising autonomy in hiring (and, therefore, 
selecting individuals who are good matches), why has the workforce not been more stable? What 
factors (including Innovation status) are influencing these high turnover rates? How can DPS support a 
more stable workforce in Innovation schools? 

In terms of the findings in this report, improvements were evident in student achievement in the first 
group of Innovation schools, but these changes cannot be attributed to Innovation status. Because rates 
of student achievement were improving in DPS overall during this time period, many reform strategies 
may be effectively improving student outcomes; Innovation status may be just one of many effective 
options. However, it may also be merely too soon to see differences between this strategy and 
competing initiatives for school reform; time is clearly a factor in the complicated process of improving 
student achievement. Examination of achievement trends over a longer period of time is warranted for 
Innovation schools in comparison to those implementing other interventions intended to raise 
achievement levels. 

If the Innovation school theory of change is accurate, improved student outcomes should be evident in 
schools where autonomy in decision-making has been exercised for that purpose. This raises additional 
questions: Have Innovation schools actually implemented changes that would require Innovation status 
(as principals reported were planned in interviews in 2011)? Additionally, if changes have been made, 
how have they been directly related to improving professional practice (as opposed to more 
organizational re-structuring around budgets, schedules, and hiring practices)? These questions must 
be answered before an expectation of improved student outcomes can be examined in a meaningful 
way and distinguished from the expectation that student outcomes improve in all schools in DPS. 
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Appendices 



Appendix A: Summary of Results from November 2011 Evaluation Report 
Methods 

The evaluation is based on a mixed-method framework which incorporates multiple sources of data. 
Data collection was framed around the Five Essential Supports for School Improvement, which have 
been found to be critical in school reform efforts by the Consortium on Chicago School Research 
(Sebring, Allensworth, Bryk, Easton, & Luppescu, 2006), and which include school structures, 
professional practice, and school cultures. 

Interviews - Between May and June of 2011 seven of the eight Innovation schools opted to 
participate in interviews. A total of seven interviews were conducted with principals, thirteen with 
teachers, and six with parents. 

Climate surveys - Between May and June of 2011, teachers at all eight of the Innovation schools 
were invited to participate in a survey to measure aspects of the climate and culture of the school. 
The total response rate across all teachers was 55% (n=347 of 626). 

Innovation Plans - An analysis of the Innovation plans submitted by schools was used to clarify 
other data collected, and to gain an understanding of what waivers each school had requested as 
part of the Innovation process. 

Achievement data - Existing data regarding student achievement on the CSAP was obtained from 
the Data Lab on the Colorado Department of Education (CDE) website. This data included the 
percent of students scoring proficient or advanced in each subject, and median growth percentiles. 

Human resources data - DPS provided evaluators with human resources files for 08-09 and 09-10 
which included information about teacher and principal staffing, experience, and degrees earned. 

Findings 

Innovation schools did not tend to look drastically different than other schools. Most principals 
reported that they have taken the approach of deliberately moving slowly with the implementation of 
their Innovation plan. It was clear that principals felt they could make more substantial changes given 
the waivers they have from district, state, and union policies; some principals were considering 
additional changes in the coming years. Given the flexibility that Innovation schools have, it is possible 
that they will begin to implement more numerous and substantive changes in the future. However, 
there remains the question of what it means for a school to be "innovative", and what expectations 
exist (from the district and the schools themselves) around what an Innovation school looks like, and 
how it may differ or not from its previous practices and from other DPS schools. 

The four major issues driving schools to seek Innovation status were: budget, schedule, workforce 
management, and level of control. These were also the areas in which respondents said they saw the 
most obvious changes following Innovation status. It was somewhat remarkable that principals did not 
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raise issues around curriculum and instruction as a key reason for seeking Innovation, though they did 
note that flexibility in these other four areas allowed them to make shifts related to instruction. The 
changes schools made which had the most potential to impact instruction had to do with teacher 
planning time, teacher PD, funds for additional academic and enrichment activities, schedule changes, 
and workforce management (e.g., hiring, opting out of direct placement). Given that there was such a 
consistent set of core drivers for all schools seeking Innovation status, this suggests there may be key 
areas where principals feel that district constraints are most burdensome. Principals appeared to view 
these issues as key levers in their management of the school; eliminating or reducing constraints in 
these areas at other DPS schools could potentially improve principals' perceptions about their ability to 
make decisions and manage their schools. At this point there is no evidence from this study to support 
or refute the idea that autonomy in these areas will improve school outcomes, but removing constraints 
in these areas clearly improved the staff and community sense of autonomy and ownership in the 
Innovation schools. 

Principals are relatively happy with the support they have received from DPS. They found that support 
improved after the formation of the Office of School Reform and Innovation (OSRI). A number of 
principals noted that in the early years of Innovation schools there had been challenges getting basic 
services from the district (particularly around HR and budget) because of a lack of understanding in 
central office. Departments did not know how to deal with the needs of Innovation schools, which 
differed from the traditional ways of doing things. Principals said that more recently, there had been a 
shift towards having dedicated central office liaisons in these departments who were knowledgeable 
about the Innovation schools and better prepared to assist. OSRI was cited by principals as a key force in 
helping the district better align systems of support for Innovation schools, though some principals felt 
that the high turnover of OSRI staff had impeded the unit's effectiveness. 

Innovation led to an increase in both real and perceived control over the schools by principals, 
teachers, and parents. This increased control was viewed as a major positive by these groups, who 
expressed a sense of greater ownership of their schools. There was a general sense of increased 
empowerment around decisions including resources, workforce, and instruction. One specific change 
that was appreciated by many respondents was the enhanced agility to make rapid decisions at the 
school level, without having to wait for approval of the decision by various central office entities. 

Having control over the workforce was a significant change in Innovation schools, from the hiring 
process to one-year contracts. The Innovation schools have made substantive changes in the way that 
they deal with their workforce. One major change was around hiring, including changes to timelines and 
the interview process. Schools were particularly pleased about opting out of direct placements; 
respondents were very negative about direct placement because they felt it led to schools having 
teachers who were not a good fit with the school's culture, philosophy, or rigor. Innovation schools 
tended to use one-year contracts with their teachers, and most respondents were happy with this. 
Parents and principals particularly liked the idea that they had a chance to determine if the teacher was 
a good fit before committing to them longer term. 

Innovation schools have experienced high rates of mobility among teachers and principals. Their 
teachers tend to be somewhat less experienced and are less likely to have master's degrees than 
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teachers in comparable schools. Although the data available for this evaluation did not allow us to draw 
strong conclusions about the effect that Innovation has on a school's workforce, these findings suggest 
that Innovation schools may have unique needs around developing and maintaining their teacher (and 
principal) workforce. 

Innovation schools tended to have overall positive cultures. Schools which had less positive cultures 
had experienced problems with the principal, principal turnover, and often lacked a clear vision for 
the school. In general, schools tended to be either high or low on all culture indicators. This suggests 
that different elements of school climate are highly intertwined, and problems in one area are likely to 
spill over into discontent in others. The interview data suggested that principal leadership was a key 
element, and that when the principal did not adequately support staff, or created an atmosphere of 
mistrust or negativity, climate indicators at the school tended to be more negative. Interestingly, the 
lack of a clear strategic vision was also present in schools which scored lowest on climate measures. 
Having a strong principal in whom the teachers and parents have trust, who is able to articulate a clear 
vision and align structures around that, seems to be an important element in the climate of schools. 

With high principal turnover at the Innovation schools, there has been some confusion about the role 
of the district in choosing a new principal. Three of the Innovation schools have changed principals 
since they gained Innovation status, and this change was associated with difficulties and discontent at 
the schools. One theme which emerged in these schools was: What is the role of the district and the 
school in choosing a new principal? There appears to be a lack of clarity around which entity will make 
the final choice. Going forward, it will be important for the district to clarify the process of principal 
hiring, and the role that staff, community, and district have in choosing a new administrator. 

Most of the Innovation schools were working on alignment across grades and subjects. Schools saw 
this work as critical, but the process was not necessarily effective at all schools. Innovation schools 
were dedicating considerable amounts of time to engaging teachers in work around creating better 
vertical and horizontal articulation. The work tended to focus on understanding what on-standard or on- 
grade work looked like. However, in several schools respondents felt the time dedicated to articulation 
was poorly used. This was generally associated with a lack of articulated goals or expectations, or a lack 
of structure to help move teachers towards the goals (since this left the onus on them to push the 
process forward). The prevalence of these articulation activities across schools raises the question of 
what support DPS provides to all district schools around operationalizing the standards, and helping 
teachers understand what on-standard and on-grade work looks like in various subjects. 

There was a lack of clarity around the boundaries of autonomy in Innovation schools - what flexibility 
they have, and what regulations they are still subject to. This theme emerged in various ways from 
principals and teachers and was centered on the idea that the district has not adopted a clear vision of 
what Innovation schools are and what they should be able to do. This has resulted in some frustration 
for school staff, who at times felt they had to battle for autonomies they thought they were entitled to 
under Innovation status, or led to confusion around district requirements. With the formation of OSRI, 
the district may now be better positioned to define the district's understandings around Innovation 
schools. However, it was clear that principals believed the district has a distance to go in defining and 
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understanding Innovation schools. It may be very useful for DPS to consider outlining the expectations 
the district has for Innovation schools in a transparent way, including the autonomies Innovation schools 
enjoy as well as the boundaries they must still adhere to. 

There are not yet clear trends to help us understand how Innovation will affect student achievement. 

Prior to gaining Innovation status, many of the Innovation schools were already trending up in terms of 
the percent of students proficient and advanced, and most also had median growth percentiles above 
the state average of 50%. 

Important questions remain about Innovation schools, and the district's role in supporting them. 

These questions are not only important in relation to the Innovation schools themselves, but are also 
critical for the district as a whole as more schools gain Innovation status. Specific questions the district 
should consider include: 

• Has DPS had a conversation around what it means to be 'innovative'? Are there particular 
expectations for what an Innovation school looks like and how it may differ or not from its 
previous practices, and from other DPS schools? 

• What does success look like for an Innovation school? Is it only about student achievement? Are 
there other factors that should be considered? (e.g., teacher satisfaction, parent involvement, 
student perceptions). 

• What are some cost implications (both in terms of additional costs or loss of economy of scale) 
as schools opt out of traditional district structures? What is the cost to the district and schools? 

• How can the district best support Innovation schools as their practices diverge from district 
offerings (e.g., around curricula, assessment, professional development, leadership, etc.)? 

• How is monitoring data about the Innovation schools used? What types of metrics are 
considered in the monitoring of Innovation schools (e.g., teacher satisfaction, parent 
involvement, teacher mobility, principal turnover, etc.)? How is this information used? 

• What supports does the district provide to assist Innovation schools who are struggling with 
various issues (e.g., principal leadership, collaboration and planning, articulation, trust, etc.? 

• The Innovation Schools Act requires a 3 year review of each school's Innovation status. What 
will be considered as part of this review? Under what circumstances would the district take 
action with regards to a school's Innovation status? Do issues like climate, student achievement, 
mobility, instruction, etc. play a role? If so, how? If not, why not? 
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Appendix B: Sampling Procedures for Comparison Schools 
STEP 1: Academic Levels 

To select other Denver Public Schools (DPS) for comparison to the Innovation Schools, evaluators first 
sorted the schools into three groups based on the academic level of students served. A preliminary 
review of possible schools showed that there were no comparable 6-12 schools although there were 
middle schools with similar demographics. To include students in the high school grades, evaluators 
decided to search for a middle school and high school within the same feeder system to replicate the 
population of 6-12 schools. The resulting groups are shown in Exhibit One. 

Exhibit One: Grouping and Goals for Comparison School Selection 



Innovation Schools 


Level 


Montclair 


Elementary 


Valdez 


Elementary 


Denver Green 


Elementary 




Cole 


K-8 


Whittier 


K-8 




Bruce Randolph 


6-12 


MLK Early College 


6-12 


Manual 


High School 



Goal - To identify 



One or more comparison elementary 
schools 



One comparison K- 8 school 



Two comparison schools - one middle 
school and one high school within a feeder 
system 



STEP 2: Free/Reduced Lunch Rates 

For each group, all other DPS schools were sorted by academic level. Within those groups, schools were 
sorted by the rate of free/reduced lunch (FRL) at those schools. Schools with FRL rates within +/■ 10 
percentage points of the average of the Innovation Schools at that level were considered as potential 
comparison schools. 

STEP 3: School Performance Framework 

For all potential comparison schools, the School Performance Framework rating (SPF) was identified. 
Schools were then matched to be consistent with the SPF rating assigned to the Innovation Schools. 

STEP 4: Tie- Breaker 

Using this process, schools were selected at the K- 8 and middle/high school levels. However, two 
elementary schools were still potential comparison schools. To decide between the two elementary 
schools, the percentage of English Language Learners was examined, and that with the closest 
percentage was selected for inclusion in the comparison group. 
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Appendix C: Climate Survey Results by Item 

This section contains the item-level analyses contrasting Innovation and Comparison schools. The scale 
for each item is indicated in the table. Items are divided into the nine scales used in this analysis. 

Three items were deleted from the final analyses because they exhibited poor scale reliabilities. These 
items were: 

• Our school would be able to perform better if we had more autonomy than we currently have to 
make decisions (e.g., scheduling, hiring, budget, instructional program). 

• Teachers at this school: feel responsible for the students they teach, but not for other students in 
the school. 



• To what extent do you feel that you have been successful in providing the kind of education you 
want to provide for your students? 

Innovation and Professional Learning 





Question text 


Mean 




Std 
Dev 


Comparison 


Teachers are willing to take risks to make this school better 


2.67 


73 


0.93 




Teachers are eager to try new ideas 


2.82 


73 


0.86 




Teachers are really working to improve their teaching 


3.07 


71 


0.74 




Teachers have a can do attitude 


2.75 


73 


0.92 




Teachers are encouraged to stretch and grow 


2.73 


73 


0.90 




Teachers are continually learning and seeking new ideas to improve their 


2.88 


72 


0.79 




practice 










Teachers respect the professional competence of their colleagues 


2.71 


73 


0.94 




Teachers seek or implement innovative strategies for improving the overall 


2.69 


72 


0.93 




quality of our school 








Innovation 


Teachers are willing to take risks to make this school better 


3.10 


152 


0.71 




Teachers are eager to try new ideas 


3.16 


151 


0.68 




Teachers are really working to improve their teaching 


3.33 


154 


0.59 




Teachers have a can do attitude 


3.25 


151 


0.63 




Teachers are encouraged to stretch and grow 


3.08 


152 


0.80 




Teachers are continually learning and seeking new ideas to improve their 


3.21 


151 


0.64 




practice 










Teachers respect the professional competence of their colleagues 


3.05 


154 


0.72 




Teachers seek or implement innovative strategies for improving the overall 


3.08 


153 


0.75 




quality of our school 








Scale: Strongly Agree (4), Agree (3), Disagree (2), Strongly Disagree (1) 
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Collaborative Environment 





Question text 


Mean 


N 


Std 
Dev 


Comparison 


Teachers discuss instructional strategies with one another 


2.96 


75 


0.86 




Teachers discuss curriculum issues with one another 


2.96 


75 


0.81 




Teachers consistently plan together 


2.59 


75 


0.89 




Teachers align their instruction with teachers at the same grade level 


2.70 


74 


0.89 




Teachers align their instruction with teachers at different grade levels 


2.39 


74 


0.87 




Teachers share a common understanding of our students' needs 


2.83 


75 


0.84 


Innovation 


Teachers discuss instructional strategies with one another 


3.09 


158 


0.65 




Teachers discuss curriculum issues with one another 


3.17 


157 


0.64 




Teachers consistently plan together 


2.66 


158 


0.84 




Teachers align their instruction with teachers at the same grade level 


2.96 


156 


0.79 




Teachers align their instruction with teachers at different grade levels 


2.48 


155 


0.75 




Teachers share a common understanding of our students' needs 


3.01 


158 


0.74 


Scale: Strongly Agree (4), Agree (3), Disagree (2), Strongly Disagree (1) 



High Quality Outcomes 





Question text 


Mean 


N 


Std 
Dev 


Comparison 


Administration has high expectations for teacher performance 
Teachers have high expectations for student performance 
Teachers have high expectations for their own performance 


2.91 
2.91 
2.95 


80 
79 
79 


1.02 
0.82 
0.81 


Innovation 


Administration has high expectations for teacher performance 
Teachers have high expectations for student performance 
Teachers have high expectations for their own performance 


3.34 
3.39 
3.43 


167 
166 
166 


0.85 
0.69 
0.70 


Scale: Strongly Agree (4), Agree (3), Disagree (2), Strongly Disagree (1) 
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Decision Making 





Question text 


Mean 


N 


Std 
Dev 


Comparison 


Decisions take into account the opinions of teachers, staff, and parent groups 


2.05 


75 


0.98 


Teachers appropriately included in decision-making 


1.93 


75 


0.93 




The principal, teachers, and staff school collaborate in effective way around school 


1.91 


75 


0.90 




decision-making 










Teachers make instructional decisions that are in the best interest of their students 


2.62 


74 


0.82 




Teachers have enough autonomy to make decisions that will positively impact their 


2.29 


75 


0.85 




students 










Teachers have autonomy to make decisions about instructional delivery 


2.51 


75 


0.84 




Decisions of school leadership about instructional practices are thoughtful and 


2.32 


74 


0.88 




aligned with our schools mission 










Decisions of our school leadership help improve our school's practices 


2.17 


75 


0.91 




Administrators have enough autonomy to make decisions and changes that will 


2.41 


74 


0.86 




benefit this school 








Innovation 


Decisions take into account the opinions of teachers, staff, and parent groups 


2.70 


161 


0.88 




Teachers appropriately included in decision-making 


2.59 


160 


0.90 




The principal, teachers, and staff school collaborate in effective way around school 


2.57 


161 


0.90 




decision-making 










Teachers make instructional decisions that are in the best interest of their students 


3.12 


161 


0.66 




Teachers have enough autonomy to make decisions that will positively impact their 


2.93 


161 


0.70 




students 










Teachers have autonomy to make decisions about instructional delivery 


3.12 


159 


0.63 




Decisions of school leadership about instructional practices are thoughtful and 


2.91 


159 


0.87 




aligned with our schools mission 










Decisions of our school leadership help improve our school's practices 


2.88 


158 


0.85 




Administrators have enough autonomy to make decisions and changes that will 


3.05 


159 


0.72 




benefit this school 








Scale: Strongly Agree (4), Agree (3), Disagree(2), Strongly Disagree (1) 
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Development of Capacity 





Question text 


Mean 


N 


Std 
Dev 


Comparison 


Take on lots of different projects at once and end up not focusing very well on 


3.05* 


80 


0.91 


any of them 










There is a lot of incoherence in the instructional programs 


2.81* 


79 


0.96 




Instructional programs are fragmented, and do not fit together very well 


2.75* 


73 


0.92 




Teachers have a shared sense of our schools mission 


2.59 


75 


0.84 




Teachers have a shared sense of what we need to do to achieve the best 


2.59 


74 


0.84 




outcomes for our students 










The educational programs are thoughtfully designed so they best meet the 


2.23 


74 


0.88 




needs of our students 










We focus on a small number of practices and try to do those very well 


2.00 


75 


0.87 




PD is differentiated to meet the needs of teachers 


2.14 


73 


0.95 




PD enhances teachers ability to implement instructional strategies to meet 


2.19 


73 


0.94 




student needs 










Professional learning opportunities in this school are aligned with the schools 


2.63 


72 


0.85 




improvement plan 








Innovation 


Take on lots of different projects at once and end up not focusing very well on 
any of them 


2.81* 


167 


0.96 




There is a lot of incoherence in the instructional programs 


2.43* 


167 


0.85 




Instructional programs are fragmented, and do not fit together very well 


2.38* 


158 


0.83 




Teachers have a shared sense of our schools mission 


3.05 


157 


0.76 




Teachers have a shared sense of what we need to do to achieve the best 


2.98 


156 


0.72 




outcomes for our students 










The educational programs are thoughtfully designed so they best meet the 


2.86 


156 


0.72 




needs of our students 










We focus on a small number of practices and try to do those very well 


2.37 


157 


0.82 




PD is differentiated to meet the needs of teachers 


2.36 


154 


0.86 




PD enhances teachers ability to implement instructional strategies to meet 


2.64 


154 


0.79 




student needs 










Professional learning opportunities in this school are aligned with the schools 


2.88 


153 


0.82 




improvement plan 








Scale: Strongly Agree (4), Agree (3), Disagree (2), Strongly Disagree (1) 








*These items were reverse coded when creating scale scores. In this table the values are presented 
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Pride and Fulfillment 





Question text 


Mean 


N 


Std Dev 


Comparison 


1 would not want to work in any other school 


2.19 


79 


1.06 


1 would recommend this school to parents seeking a place for their 


2.38 


79 


0.96 




child 










1 would send my own child to this school 


2.12 


78 


1.03 




1 usually look forward to each working day 


2.78 


79 


0.86 




1 feel loyal to this school 


2.73 


79 


0.97 




1 feel proud being a teacher at this school 


2.74 


78 


0.97 


Innovation 


1 would not want to work in any other school 


2.63 


163 


0.96 




1 would recommend this school to parents seeking a place for their 


2.85 


163 


0.93 




child 










1 would send my own child to this school 


2.49 


161 


1.03 




1 usually look forward to each working day 


2.99 


163 


0.79 




1 feel loyal to this school 


3.15 


163 


0.84 




1 feel proud being a teacher at this school 


3.20 


163 


0.85 


Scale: Strongly Agree (4), Agree (3), Disagree (2), Strongly Disagree (1) 



Accountability 





Question text 


Mean 


N 


Std Dev 


Comparison 


Teachers feel responsible for helping all students who are struggling 


3.13 


79 


0.82 




Teachers feel responsible for helping other teachers do their best 


2.77 


79 


0.93 




Teachers feel responsible for helping improve the entire school 


2.86 


79 


0.96 




Teachers feel responsible for helping all students learn 


3.09 


77 


0.88 




Teachers are committed to helping students learn 


3.24 


79 


0.75 


Innovation 


Teachers feel responsible for helping all students who are struggling 


3.43 


164 


0.68 




Teachers feel responsible for helping other teachers do their best 


3.05 


163 


0.79 




Teachers feel responsible for helping improve the entire school 


3.21 


164 


0.68 




Teachers feel responsible for helping all students learn 


3.40 


163 


0.67 




Teachers are committed to helping students learn 


3.48 


163 


0.61 


Scale: Strongly Agree (4), Agree (3), Disagree (2), Strongly Disagree(l) 
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Empowerment 





Ouestion text 


Mean 


N 


Std 
Dev 


Comnarison 

CC 1 1 1 KJ CI 1 1 JU 1 1 


Tpachpr influpncp* Studpnt bphavior nolicips ^ 

1 ^.Q^l 1^. 1 II II 1 V_^l V — - 1 1 V_- C • JLUU^I 11 KJ CI 1 CI V 1 C 1 Kj \J 1 1 C 1 C *J 


2.06 


84 


1.01 




Teacher influence: Planning school budgets b 


1.70 


83 


0.87 




Teacher influence: Determining their own teaching assignments b 


2.42 


83 


1.01 




Tpachpr influpncp* Dptprmininf? thp school's schpdulp b 

1 C CI C 1 1 V — - 1 II II 1 UC 1 1 vL i I—/ C LC 1 Mill III 1 ^ LI IC Jvl 1 w V-/ 1 %J JL-I ICU U IC 


1.89 


84 


0.98 




Tpachpr influpncp* Dptprminin? thp school's curriculum b 

1 C CI C 1 1 C 1 II II IC^, ■ L-/ C LC 1 Mill III 1 & LI IC JCI 1 vv vv 1 _) C C4 1 1 1 L>U 1 U 1 1 1 


1.81 


84 


0.95 




Tpachpr influpncp* Hiring npw tpachprs ^ 

1 CQCI Ic 1 II II 1 UU 1 ICC. 1 III ' ' ' 1 1 C vv LCQLI Ic 1 J 


2.16 


83 


0.86 




Control over decisions: Selecting textbooks c 


1.76 


84 


0.91 




Control ovpr dpcisions* Splpctinf? othpr instructional matprials c 

C w 1 1 L 1 V-/ 1 V-/ V C 1 v( C C U 1 C* 1 1 Ji <JC ICLyLl 1 l& C* LI 1 C 1 II IJ LI UL>LI C/ IICII 1 1 IU LC 1 IU IJ 


2.49 


84 


1.00 




Control over decisions: Selecting the content, topics, and skills to be taught c 


2.30 


83 


0.96 




Control over decisions: Selecting the teaching techniques that you will use c 


2.80 


84 


0.93 




Control over decisions: Selecting how to assess your students' learning c 


2.45 


84 


0.92 




Adults who work most closely with students best positioned to make decisions to 


3.40 


80 


0.72 




benefit those students a 










Teachers are empowered to make instructional decisions in their classrooms 3 


2.44 


80 


0.90 




Teachers believe that all students are competent learners a 


2.71 


73 


0.94 




Teachers believe that every student is capable of learning and improving a 


2.86 


73 


0.90 


Innovation 

1 1 1 1 1 CV V U 11 C 1 1 


Tpachpr influpncp* Studpnt bphavior nolicips 

1 C CI C 1 ICI II II 1 v>C ■ JLUUL 1 1 L KJ CI 1 CI V 1 C 1 UJ Lv 1 1 C 1 C <J 


2.73 


175 


0.86 




Tpachpr influpncp* Planning school huri^pts 


1.85 


172 


0.91 




Teacher influence: Determining their own teaching assignments 


2.71 


174 


0.94 




Tpachpr influpncp* Dptprminin? thp school's schpdulp 

1 C CI C 1 1 C 1 II 1 1 1 d C 1 ILyC « LV C LCI Mill III 1 g LI IC JCI 1 CV CV 1 J> JL>I ICU U IC 


2.14 


175 


0.97 




Tpachpr influpncp* Dptprminin? thp school's curriculum 

1 CQ C 1 ICI II II 1 U C 1 ICC. LV C LCI Mill III 1 g LI IC Jv_l 1 C C 1 J> C CI 1 1 1 CU 1 U 1 1 1 


2.59 


176 


1.00 




Tpachpr influpncp* Hirin? npw tpachprs 

1 CUCI IL 1 II II IUCI ILCi 1 III II Ig 1 IC VV LCQLI ICI J 


2.70 


176 


0.97 




Control ovpr dpcisions* Splpctins? tpxtbooks 

C-C* 1 1 L 1 C/ 1 C/ V C 1 \A C vl JI C* 1 1 J ■ JCICLLII Ig LC/\L UL/W IX *J> 


2.41 


176 


1.00 




Control ovpr dpcisions* Splpctin? othpr instructional matprials 

CC 1 1 L 1 C 1 C V C 1 UCLIJI C 1 IJ ■ JCICLLI 1 Ik C LI ICI II 1 JLI UCLI C IICII 1 1 IU LCI IU 1 J 


2.95 


175 


0.90 




Control over decisions: Selecting the content, topics, and skills to be taught 


2.81 


174 


0.93 




Control over decisions: Selecting the teaching techniques that you will use 


3.29 


174 


0.79 




Control over decisions: Selecting how to assess your students' learning 


2.94 


176 


0.79 




Adults who work most closely with students best positioned to make decisions to 


3.35 


167 


0.63 




benefit those students 










Teachers are empowered to make instructional decisions in their classrooms 


3.13 


167 


0.74 




Teachers believe that all students are competent learners 


3.22 


154 


0.70 




Teachers believe that every student is capable of learning and improving 


3.29 


154 


0.70 


a Scale: Strongly Agree (4), Agree (3), Disagree (2), Strongly Disagree (1) 








b Scale: Great Extent (4), Some Extent (3), A little bit (2), Not at all(l) 








c Scale: Complete control (4), Moderate control (3), A little bit of control (2), No control (1) 
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Ownership 











Std 




Question text 


Mean 


N 


Dev 


Lomparison 


i Teei a strong sense ot ownersnip aoout wnai nappens in my classroom 


D.lO 


oU 


U. to 




1 feel a strong sense of ownership about what happens in my school 


2.04 


80 


0.97 




Culture of ownership, where teachers believe they are responsible for 


2.21 


80 


0.87 




outcomes of all students 








Innovation 


1 feel a strong sense of ownership about what happens in my classroom 


3.56 


167 


0.65 




1 feel a strong sense of ownership about what happens in my school 


2.67 


166 


0.95 




Culture of ownership, where teachers believe they are responsible for 


2.99 


166 


0.85 




outcomes of all students 








Scale: Strongly Agree (4), Agree (3), Disagree (2), Strongly Disagree (1) 
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Appendix D: Discussion Questions for Climate Survey Results 

• Which scales are the ones prioritized as areas where Innovation schools should receive higher 
ratings? Are these in fact the scales that received higher ratings? 

• Which scales are lower than wanted? 

• Given how central concepts of Empowerment and Ownership are in Innovation schools, are the 
differences between the Innovation and Comparison schools as large as expected? Are levels of 
Empowerment and Ownership as high in Innovation schools as expected? 

• Innovation schools scored lower on Capacity than on any other scale, yet Capacity had been 
discussed by stakeholders as an important foundational component of a school's ability to 
meaningful enact Innovation. Given the items on the Capacity scale, how can we interpret 
Innovation schools' scores on this scale? 

• If we consider individual items, which items suggest that things are going very well in the 
Innovation/Comparison schools? 

• If we consider individual items, which items suggest that there are areas of challenge in the 
Innovation/Comparison schools? 

• For items which are lower than wanted, are these important issues for Innovation schools? 
What might be causing some of the challenges? What additional information is needed? What 
could DPS do in order to help mitigate some of these challenges? 

• Looking at items which measure actual behavior, not just perception (for example, review items 
on the Collaborative Environment, Empowerment, or Decision Making subscales). 

o How different are practices in the Innovation schools and the Comparison schools? 
o Are teacher responses in the Innovation schools as high as wanted? 
o Which behaviors are most important? Are Innovation teachers reporting that these 
happen at high levels? 

o Which important behaviors are not happening as much as wanted? What role might DPS 
have in helping promote these behaviors going forward? 
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Appendix E: Teacher Turnover Rates by Year 



Percentages of teachers who left the school as compared to number employed in prior year: 



School Year 


Innovation 
Cohort One 


Innovation 
Cohort Two 


Comparison 


DPS 


2007-08 


36.5 


27.5 


34.1 


21.3 


2008-09 


33.0 


29.7 


22.9 


18.4 


2009-10 


23.3 


20.0 


11.4 


14.1 


2010-11 


22.7 


21.9 


9.8 


16.6 


2011-12 


27.7 


46.4 


24.2 


16.6 
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Appendix F: Innovation TCAP/CSAP Results, by Cohort and Calendar Year 



Percent Proficient/Advanced in Math for the 2009 cohort 




— i 1 1 1 1 1 1 1 1 — 

03-04 04-05 05-06 06-07 07-08 08-09 09-10 10-11 11-12 

Academic Year 
School • • • Bruce Randolph School 
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Percent Proficient/Advanced in Reading for the 2009 cohort 




n 1 1 1 r 

03-04 04-05 05-06 06-07 07 



1 1 1 r 

09 09-10 10-11 11-12 



School 



Academic Year 
* Bruce Randolph School 
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Percent Proficient/Advanced in Writing for the 2009 cohort 




— i 1 1 1 1 1 1 1 1 — 

03-04 04-05 05-06 06-07 07-08 08-09 09-10 10-11 11-12 



Academic Year 
School • • • Bruce Randolph School 
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Median Growth Percentile in Math for the 2009 cohort 




— i 1 1 1 1 1 1 1 1 — 

03-04 04-05 05-06 06-07 07-08 08-09 09-10 10-11 11-12 



Academic Year 
School • • • Bruce Randolph School 
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Appendix H: ACT Results by Cohort Relative to Innovation Status 
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